Defining and characterising structural uncertainty in decision analytic models

An inappropriate structure for a decision analytic model can potentially invalidate estimates of cost-effectiveness and estimates of the value of further research. However, there are often a number of alternative and credible structural assumptions which can be made. Although it is common practice to acknowledge potential limitations in model structure, there is a lack of clarity about methods to characterize the uncertainty surrounding alternative structural assumptions and their contribution to decision uncertainty. A review of decision models commissioned by the NHS Health Technology Programme was undertaken to identify the types of model uncertainties described in the literature. A second review was undertaken to identify approaches to characterise these uncertainties. The assessment of structural uncertainty has received little attention in the health economics literature. A common method to characterise structural uncertainty is to compute results for each alternative model specification, and to present alternative results as scenario analyses. It is then left to decision maker to assess the credibility of the alternative structures in interpreting the range of results. The review of methods to explicitly characterise structural uncertainty identified two methods: 1) model averaging, where alternative models, with different specifications, are built, and their results averaged, using explicit prior distributions often based on expert opinion and 2) Model selection on the basis of prediction performance or goodness of fit. For a number of reasons these methods are neither appropriate nor desirable methods to characterize structural uncertainty in decision analytic models. When faced with a choice between multiple models, another method can be employed which allows structural uncertainty to be explicitly considered and does not ignore potentially relevant model structures. Uncertainty can be directly characterised (or parameterised) in the model itself. This method is analogous to model averaging on individual or sets of model inputs, but also allows the value of information associated with structural uncertainties to be resolved.

Bridging the gap between methods research and the needs of policy makers: A review of the research priorities of the National Institute for Health and Clinical Excellence

Objectives: The aim of this study was to establish a list of priority topics for methods research to support decision making at the National Institute for Health and Clinical Excellence (NICE). Methods: Potential priorities for methods research topics were identified through a focused literature review, interviews, an email survey, a workshop and a Web-based feedback exercise. Participants were members of the NICE secretariat and its advisory bodies, representatives from academia, industry, and other organizations working closely with NICE. The Web exercise was open to anyone to complete but publicized among the above groups. Results: A list of potential topics was collated. Priorities for further research differed according to the type of respondent and the extent to which they work directly with NICE. Priorities emerging from the group closest to NICE included: methodology for indirect and mixed treatment comparisons; synthesis of qualitative evidence; research relating to the use of quality-adjusted life-years (QALYs) in decision making; methods and empirical research for establishing the cost-effectiveness threshold; and determining how data on the uncertainty of effectiveness and cost-effectiveness data should be taken into account in the decision-making process. Priorities emerging from the broadest group of respondents (through the Web exercise) included: methods for extrapolating beyond evidence observed in trials, methods for capturing benefits not included in the QALY and methods to assess when technologies should be recommended in the context of further evidence gathering. Conclusions: Consideration needs to be given to the needs of those who use the outputs of research for decision making when determining priorities for future methods research.NIHR Medical Research Council

Laparoscopic fundoplication compared with medical management for gastro-oesophageal reflux disease : cost effectiveness study

Peer reviewedPublisher PD

Reflecting Parameter Uncertainty in Addition to Variability in Constrained Healthcare Resource Discrete Event Simulations : Worth Going the Extra Mile or a Road to Nowhere?

Objectives Probabilistic sensitivity analysis (PSA) has been shown to reduce bias in outcomes of health economic models. However, only 1 existing study has been identified that incorporates PSA within a resource-constrained discrete event simulation (DES) model. This article aims to assess whether it is feasible and appropriate to use PSA to characterize parameter uncertainty in DES models that are primarily constructed to explore the impact of constrained resources. Methods PSA is incorporated into a new case study of an Emergency Department DES. Structured expert elicitation is used to derive the variability and uncertainty input distributions associated with length of time taken to complete key activities within the Emergency Department. Potential challenges of implementation and analysis are explored. Results The results of a trial of the model, which used the best estimates of the elicited means and variability around the time taken to complete activities, provided a reasonable fit to the data for length of time within the Emergency Department. However, there was substantial and skewed uncertainty around the activity times estimated from the elicitation exercise. This led to patients taking almost 3 weeks to leave the Emergency Department in some PSA runs, which would not occur in practice. Conclusions Structured expert elicitation can be used to derive plausible estimates of activity times and their variability, but experts’ uncertainty can be substantial. For parameters that have an impact on interactions within a resource-constrained simulation model, PSA can lead to implausible model outputs; hence, other methods may be needed

Extrapolating Survival from Randomized Trials Using External Data: A Review of Methods.

This article describes methods used to estimate parameters governing long-term survival, or times to other events, for health economic models. Specifically, the focus is on methods that combine shorter-term individual-level survival data from randomized trials with longer-term external data, thus using the longer-term data to aid extrapolation of the short-term data. This requires assumptions about how trends in survival for each treatment arm will continue after the follow-up period of the trial. Furthermore, using external data requires assumptions about how survival differs between the populations represented by the trial and external data. Study reports from a national health technology assessment program in the United Kingdom were searched, and the findings were combined with "pearl-growing" searches of the academic literature. We categorized the methods that have been used according to the assumptions they made about how the hazards of death vary between the external and internal data and through time, and we discuss the appropriateness of the assumptions in different circumstances. Modeling choices, parameter estimation, and characterization of uncertainty are discussed, and some suggestions for future research priorities in this area are given

Eliciting uncertainty for complex parameters in model-based economic evaluations: quantifying a temporal change in the treatment effect

Background. In model-based economic evaluations, the effectiveness parameter is often informed by studies with a limited duration of follow-up, requiring extrapolation of the treatment effect over a longer time horizon. Extrapolation from short-term data alone may not adequately capture uncertainty in that extrapolation. This study aimed to use structured expert elicitation to quantify uncertainty associated with extrapolation of the treatment effect observed in a clinical trial. Methods. A structured expert elicitation exercise was conducted for an applied study of a podiatry intervention designed to reduce the rate of falls and fractures in the elderly. A bespoke web application was used to elicit experts’ beliefs about two outcomes (rate of falls and odds of fracture) as probability distributions (priors), for two treatment options (intervention and treatment as usual) at multiple time points. These priors were used to derive the temporal change in the treatment effect of the intervention, to extrapolate outcomes observed in a trial. The results were compared with extrapolation without experts’ priors. Results. The study recruited thirty-eight experts (geriatricians, general practitioners, physiotherapists, nurses, and academics) from England and Wales. The majority of experts (32/38) believed that the treatment effect would depreciate over time and expressed greater uncertainty than that extrapolated from a trial-based outcome alone. The between-expert variation in predicted outcomes was relatively small. Conclusions. This study suggests that uncertainty in extrapolation can be informed using structured expert elicitation methods. Using structured elicitation to attach values to complex parameters requires key assumptions and simplifications to be considered

Cost-effectiveness of a proportionate universal offer of free exercise : Leeds Let’s Get Active

Objectives To assess the cost-effectiveness of a proportionate universal programme to reduce physical inactivity (Leeds Let’s Get Active) in adults. Methods A continuous-time Markov chain model was developed to assess the cost implications and QALY gains associated with increases in physical activity levels across the adult population. An ordered logistic model was specified to estimate the effectiveness of the Leeds Let’s Get Active programme and derive transition probabilities between physical activity categories. A parametric survival analysis approach was applied to estimate the decay of intervention effect over time. Baseline model data were obtained from previous economic models, population-based surveys and other published literature. A cost-utility analysis was conducted from a health care sector perspective over the programme duration (39 months). Scenario and probabilistic sensitivity analyses were performed to test the robustness of cost-effectiveness results. Results 51,874 adult residents registered to the programme and provided baseline data, 19.5% of which were living in deprived areas. Under base case assumptions, Leeds Let’s Get Active was found to be likely to be cost-effective. However, variations in key structural assumptions showed sensitivity of the results. Conclusions Evidence from this study suggests that a universal offer of access to free off-peak leisure centre-based exercise that targets hard to reach groups can provide good value for money. Further data collection is needed to reduce the uncertainty surrounding the decision